Object - Based Digital Video Processing Using 2 - D Meshes

This thesis addresses extraction and tracking of video objects for object-based representation and manipulations of synthetic and natural digital video. Object-based video representation has gained a signi cant momentum by recent developments in multimedia computing, and has emerged as a major area of research. Many multimedia applications, such as bit-stream editing, interactive TV, and digital libraries demand object-based video representations. Furthermore, the ongoing MPEG-4 standardization e ort is aimed at video compression methods that support object-based functionalities. Objects are de ned as semantically meaningful components of a scene. For example, in sports, outdoor and indoor scenes, objects are to be likely billboards, people and furniture, respectively. Object tracking refers to tracking the boundary, local motion and intensity (contrast and brightness) variations of an object. Object tracking is essential to semi-automatic object extraction and object-based video representation. Digital video manipulation refers to synthesis of new video content by combining natural and synthetic sources, such as graphics, animation, still images, and moving images. Two examples that are addressed in this thesis are synthetic object trans guration and augmented reality. We propose a new approach to object tracking using forward 2-D mesh tracking and object mosaicing. We unify the problem of tracking the boundary, local motion and intensity (contrast and brightness) variations of an object using 2-D object based meshes. We present a hierarchical 2-D mesh tracking method (without the need for long sequences) and a new method to propagate the mesh structure from one frame to another without the need for motion estimation at each node of the mesh. We suggest novel convexity constraints into the mesh re nement procedure to preserve the connectivity of the mesh and a method for compensating intensity variations based on a spatially continuous model. The results clearly demonstrate the importance of local compensation of intensity variations for accurate object tracking. Furthermore, we extend our mesh tracking method to tracking the objects with occlusion. We suggest to use dense motion estimation to handle the problems